The INEX Evaluation Initiative

The widespread use of the extensible Markup Language (XML) on the Web and in Digital Libraries brought about an explosion in the development of XML tools, including systems to store and access XML content. As the number of these systems increases, so is the need to assess their benefit to users. The benefit to a given user depends largely on which aspects of the user’s interaction with the system are being considered. These aspects, among others, include response time, required user effort, usability, and the system’s ability to present the user with the desired information. Users then base their decision whether they are more satisfied with one system or another on a prioritised combination of these factors. The Initiative for the Evaluation of XML Retrieval (INEX) was set up at the beginning of 2002 with the aim to establish an infrastructure and provide means, in the form of a large XML test collection and appropriate scoring methods, for the evaluation of content-oriented retrieval of XML documents. As a result of a collaborative effort, INEX created an XML test collection consisting of publications of the IEEE Computer Society, 60 topics and graded relevance assessments. Using the constructed test collection and the developed set of evaluation metrics and procedures, the retrieval effectiveness of the participating organisations’ XML retrieval approaches were evaluated and their results compared [4]. In this chapter we provide an overview of the INEX evaluation initiative. Before we talk about INEX, we first take a brief look, in section 2, at the evaluation practices of information retrieval (IR) as these formed the basis of our work in INEX. In our discussion of INEX we follow the requirements that evaluations in IR are founded upon [10]. These include the specification of the evaluation objective (e.g. what to evaluate) in section 3, and the selection